Designing an effective questionnaire

(Internal CYC Document based on LGA/LA Guidance)

This document does not cover BI Guidelines around electronic surveys, nor accessibility. This guidance provides you with hints and tips on:

· The design and structure of your questionnaire

· Question types

· What can have a negative impact

Before you start writing your questionnaire it is important that you are clear about the main aims and objectives. These should then enable you to develop your questionnaire more easily and ensure that you are asking questions that will give you the information that you need. The structure of your questionnaire doesn’t have to follow the same order as the aims and objectives but should be logical from the respondent’s point of view. Use the following additional hints and tips to help you design an effective questionnaire:

· In practice, there is no right or wrong question or answer. Your questions should be linked to the data you need to obtain. Bear in mind that different questions can get different answers.

· Whenever possible, a questionnaire should follow some form of natural flow, reflecting a train of thought, a logical conversation, or a sequence of events, depending on the subject matter.

· Include a simple introduction – this can help encourage participation. In this you should tell the respondent:

-         who you are

-         the purpose of your survey and why you want their input

-         a reason why they should take part

-         how long it will take them

-         terms of anonymity and confidentiality

-         how the results will be used

-         when and how they will receive feedback

-         who they can contact if they have for any queries

· It is always best to start with a simple question. Sensitive topics should be explored through appropriately sensitive or even indirect questions and are best to be placed towards the end of the survey.

· Group similar questions together – you may want to include section headings if appropriate.

· Questions should be relevant, concise, clear and efficient. Most people make the mistake of asking too many and questions that they will not use. You want to get a good response so clear and concise questions will help to achieve this. Be really strict when thinking about your questions – ask yourself how will this question add insight or understanding?

· Only one question should be asked at a time- if you ask two questions in one sentence then you will be unclear about what an answer is referring to.

· Care should be taken to ensure that respondents are not lead to answer in a particular way. For example, any facts to be communicated to the respondent should be set out in a neutral manner. Positive phrases such as “the proposal is a good idea” in a question without a balancing negative phrase may lead the respondent to a positive view.

· Think about the type of questions you are asking. Are you asking questions that can be answered:

-    Accurately (What is your age?)

-     Through memory (e.g. how many times in the past 12 months have you visited....?)

-    Through a best choice of options, none of which may correspond precisely to the respondent’s view/behaviour (e.g. Which of these three statements comes closest to describing your view...?)

-    Through estimation, guesswork or even speculation

· Appropriate answers should be available to reflect the reality of the range of responses. Available answers should be clear and understandable, with little ambiguity as possible and should be phrased to the respondent’s way of thinking/talking.

· Don’t know or No view should be available to the respondent as it is often a perfectly valid answer, and a “not answered” response. You may also need to include a not applicable if necessary.

· Where it is difficult to predict the full range of answers, you could provide an ‘other or please specify’ category with a box to enter free text.

· Where appropriate, consider using questions from previous research as this gives comparability and can enhance the value of the data collected.

· Ask yourself, will your respondents understand your questions and options, be willing to answer your questions and able to answer them.

· Always pilot your questionnaire either with colleagues or people from your target audience. The aim here is to detect any flaws in your questioning and correct these prior to the main survey. This may also help you to change some open questions to questions with options. You may also be able to do a trial analysis on your sample.

· Questionnaires should be formally agreed by all interested parties. It is important that everyone agrees that the aims and objectives will be sufficiently met by the questions asked.

Question types

The type of questions used in your questionnaire play a key role in producing unbiased or relevant responses and the quality of data you receive back. Question types range from open-ended (asking for comments) to closed-ended (yes/no). These are described in more detail below.

1) Open-Ended

Open –ended questions allow people to answer a question in their own words. We want to ensure that people have an opportunity to express their views and feelings and open text boxes can allow respondents to raise issues that you might not of thought of, however, think carefully about your consultation and what you are trying to find out as sometimes respondents might find it difficult to express their feelings, it can take longer for the respondent to complete and analysing comments can be time consuming. The best way to collect comments is by using text boxes as opposed to using lines or a clear space.

2) Closed-Ended

Closed-ended questions include pre-designed answers with a small or large set of choices. When providing a set of choices it is important to cover all the possible answers that are expected. You may also want to include an ‘other’ category or comments box as it can be difficult to pre-determine all of the possible answers. There are different types of closed questions that you might want to use. This includes:

3) Multiple Choice

This type includes a set of pre-designed answers, for example yes/no/don’t know or once/twice/three times).

4) Ranked questions

This type of question can be used when all the choices listed need to be ranked, for example you might ask people to rank how important services are from 1^st to 5^th Choice. It is important to pilot these questions to ensure that they are clear and will be completed correctly.

5) Rating questions

This type of question can be used when you want to measure people’s ideas, opinions, knowledge or feelings, for example you might ask people to what extent they agree or disagree with something as demonstrated below:

6) Balanced vs. unbalanced rating scales

A five-point rating scale is usually sufficient and easily understood by people. However, there is no set limit on the number of categories to use. Using too few could give less useful information and too many could make it hard to answer. It is better to provide a balanced scale as it means that an equal number of positive and negative options are included which helps to avoid bias.

Example 1 – balanced

Excellent       Good              Average         Poor                Very Poor

This example includes two positives, two negatives and a neutral midpoint.

Example 2 – unbalanced

Excellent       Very good      Good              Average         Poor

There are some occasions in which an unbalanced scale is suitable. For example in a customer satisfaction survey, few customers may say that something is unimportant. In the example below ‘important’ will become the midpoint.

· Extremely important

· Very important

· Important

· Neither important nor unimportant

· Not important

7) Not applicable (N/A) / Don’t know / No view

It is advisable to allow people to opt out if they don’t know/have no view or if it does not apply to them as it will help increase the response rate and quality of data collected.

The following aspects can have a negative impact on your consultation:

· Excessively lengthy questionnaires – consider whether a question is required to meet the aims and objectives, or whether it is “nice to know”.

· Lack of honesty/transparency

· Excessive “about you” section – personal data should be relevant

· Repetitive questioning

· Lack of opportunity for respondents to have their say – when asking closed questions you may also want to ask ‘why do you say that’ or ‘provide further comments below’. This encourages respondents to feel that the council is interested and values their views. It may prove beneficial to the consultation as it may cover aspects that are not covered in the questionnaire, provide other opinions and give a breadth of views.